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The pandemic has caused almost all educational institutions to use online 
learning media to support learning activities. E-learning is a technology that 
is widely used because it can accommodate all learning activities. However, 
in general, e-learning can only perform automatic assessments for multiple 
choice answers but not for essay answers, so that manual assessment by the 
teacher becomes difficult and takes a long time. In this study, the winnowing 
algorithm was applied to the automatic assessment process on students' essay 
answers by measuring their similarity to the teacher's answer key. The stages 
in the automatic assessment using the winnowing algorithm begin with 
forming a series of k-grams, calculating the hash value, forming a window 
from the hash value, calculating the fingerprint value, and calculating the 
Jaccard Coefficient to obtain the percentage of text similarity results. The test 
results show that the winnowing algorithm can provide good performance 
when the answers to questions are in the form of short entries with the number 


of hashes not smaller than the window value. Meanwhile, on questions with 
long answers, the winnowing algorithm can still work well with an average 
difference of 5.2% from the results of the assessment carried out by the 
teacher. 
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1. INTRODUCTION 

The use of e-learning in the learning process is a form of adjustment to the development of information 
technology. Several educational institutions have developed e-learning systems with various features that can 
support the ease of the learning process since before the pandemic until now [1]. With e-learning, the 
distribution of material from teachers and students can be easily carried out and coordinated. In addition, 
students can also collect assignments and work on exam questions on the e-learning platform. In essence, e- 
learning can be used by students and teachers to carry out all learning activities integrated into a system [2]— 
[4], so that e-learning makes the activities of the learning process more efficient, creates a pleasant learning 
experience for students, and increases their learning activities [5]-[7]. 

The e-learning existence makes it easy for teachers to evaluate student learning outcomes through 
taking exams or assignments. Exam questions can be arranged in various forms, such as multiple-choice, true 
or false, short answers, and essays. Of course, the mechanism for assessing student learning outcomes adjusts 
to the type of questions given to students. Essay questions are considered to be the most helpful technique for 
measuring student learning outcomes. Essay questions are considered as a measuring tool for students’ ability 
to memorize, remember, analyze, and write the results of their thoughts [8]. With essays, students can 
demonstrate their skills and knowledge related to the learning material that has been received [9]. However, 
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the assessment of essay answers is quite difficult for teachers to do because the process takes a lot of time, 
especially if the number of questions and students in a class is quite large, causing more questions to be assessed 
by the teacher. This condition can cause the quality of the assessment to decrease, and it is difficult to make an 
objective assessment [10]. 

The automatic assessment feature can be applied to e-learning to overcome these problems. Thus, the 
assessment of student answers in the form of essays can be done by the system. Several techniques have been 
applied by previous researchers related to this problem, such as Rabin Karp, latent semantic analysis (LSA), 
and Cosine Similarity. Rabin Karp is an algorithm that can be used to measure the level of similarity of text. 
This algorithm can be used to check the plagiarism level of a document [11]. Rabin Karp works by matching 
strings using a hash function as a comparison between search strings and substrings in the text. If the values of 
the two hashes are the same, then a further comparison of the characters will be carried out [12]. However, the 
weakness of Rabin Karp is its inability to perform a single pattern search process. The hash calculation on 
Rabin-Karp only calculates the number of hashes that have the same value in both documents [13]. Thus, if 
answers have the same meaning but the choice of words is not the same, it will produce a low level of similarity. 

LSA as one of the techniques that can be used to carry out automatic assessments is a relatively simple 
method but has a reasonably high level of correlation [14]. Besides being used for automatic assessment [14], 
[15], LSA can also be applied to check plagiarism [16] and text summaries [17], [18]. Unfortunately, the LSA 
only assesses these documents by the terms in each document, so the word order or the layout of the terms is 
not taken into account. Even though this indirectly also affects the meaning contained in each document [19]. 
Another technique that is also often used to perform automated research on essay answers is Cosine Similarity. 
Cosine Similarity provides competitive results, as well as lower levels of complexity and faster load times 
compared to LSA [20]. As one of the popular methods to be applied to the calculation of document similarity, 
Cosine Similarity cannot be affected by the length or shortness of a document because the term value of each 
document is considered principal [21], [22]. Cosine Similarity has proven to be successful in conducting 
automatic essay assessments using N-Gram [23] and WordNet [24]. 

Another algorithm that is often used to solve problems related to text processing is winnowing. 
Assessment of essay answers can be done automatically by matching the strings of each word being compared. 
This algorithm can increase the efficiency of the document comparison process because of the concept of a 
window so that a hash value is formed, which is then used in the document matching process. In addition, the 
winnowing algorithm is also proven to provide a much greater accuracy value than Rabin Karp and Knuth 
Morris Pratt when used to detect plagiarism in documents [25]. 

The relationship between the parameters in the winnowing algorithm, such as k-grams, windows, and 
prime numbers, with the level of similarity, has not been clarified, so it is still unclear what effect the value 
will have on the results of the similarity measurement given. Therefore, the purpose of this study is to apply 
the winnowing algorithm with various combinations of parameter values to determine its impact on the results 
of automatic assessment of student answers in e-learning. By developing the automatic assessment feature in 
e-learning, objectivity and time efficiency in assessing essay answers are expected to be improved. Thus, the 
proposed e-learning system can accommodate various features that can make it easier for teachers and students 
to carry out the online learning process during the pandemic. In addition, it is easier for teachers to evaluate 
student learning outcomes in the form of assessment assignments, quizzes, or student exams that are carried 
out automatically in e-learning. 


2. NOVELTY 

The COVID-19 pandemic has had a major impact on the world of education. Learning from home 
using online media is a solution so that the teaching and learning process continues to run well. Adequate 
distance learning (e-learning) will provide great opportunities for students and educators to acquire knowledge 
and improve skills. E-learning can be defined as a computer-based online learning process used by students to 
review lecture materials, discussion forums, lecturer assessments, exam questions without reducing class face- 
to-face time [26]. E-learning can be presented in various forms of text, sound, video, images, and animation. 
E-learning can be accessed by students regardless of time and place [27], thus giving students more 
opportunities to repeat the existing material [28]. However, generally, e-learning developed by several previous 
researchers only had standard features consisting of discussion forums, uploading and downloading files, email 
notifications, student progress reviews, and search features [29]. 

This study offers e-learning in a single learning media portal to support and facilitate the learning 
process in Vocational High Schools. The e-learning features upload/download materials, online 
chat/discussion, automatic assessment of student evaluations (multiple choice and essay), real-time reporting 
of student grades, and attendance. Upload/download this material will be differentiated by subject and major, 
so students can only access those according to their major. Online chat/discussion is used to make it easier for 
students and teachers to discuss in real-time during learning hours according to the material at the meeting. 
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This automatic assessment of student evaluation is used to make it easier for teachers to evaluate multiple 
choice answers or essays automatically. In addition to the multiple-choice automatic assessment model, in this 
case, the researcher also focuses on essay questions. The automatic essay answer scoring system is a way that 
can facilitate and speed up the process of assessing essay answers so that teachers do not need to make 
corrections manually. Thus, it becomes easier for teachers to evaluate, specifically when assessing student 
practicum reports. 


3. METHOD 

In this study, the automatic assessment process for essay answers was carried out by implementing 
the winnowing algorithm. Winnowing algorithm is a method of finding similar words in documents by 
comparing fingerprints on documents [30]. The implementation of the winnowing algorithm requires input in 
the form of an answer text, then it will produce an output in the form of a percentage of the results of the 
similarity of the text. Figure 1 is a flowchart of the automatic assessment process for essay answers using the 
winnowing algorithm. 


Calculation of Hash Value 
Similarity 


Figure 1. Flowchart of the proposed method 


Based on Figure 1, text input is a character string that will be used as input to be processed in the 
winnowing algorithm. This input will be matched with the answer key text document to obtain a level of 
similarity, which will generate the correct answer value for the answer key automatically. The input of this 
algorithm is a text document that is processed to produce output in the form of a collection of hash values 
called fingerprints [31]. This fingerprint is used as a basis for comparison between the entered text file and the 
teacher's answer key so that the level of similarity of the text is obtained. 
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3.1. Formation of k-Gram series 

In the winnowing algorithm, there is a k-gram formation which is a method for separating text by 
forming a substring along with k characters from a string [32]. The k-Gram is a series of terms with length k. 
The formation of the k-Gram series in the winnowing algorithm is done by forming a series of characters along 
with k from the results of removing irrelevant characters. A good k value is neither too small nor too large. The 
first k-Gram series starts from the lst character to the k-th character, the second series starts from the 2nd 
character to the k+1 character, and so on until a k-Gram series of all characters is formed. Figure 2 shows an 
example of forming a k-Gram series in the text “bunga merah” (without quotes) with k=3. 


Figure 2. Formation of k-gram on "bunga merah" 


With k=3 in the text “bunga merah”, eight k-Grams are formed as: 
= [st k-Gram_ : bun 
- 2nd k-Gram : ung 
= 3rd k-Gram : nga 
- 4th k-Gram : gam 
* Sth k-Gram : ame 
- 6th k-Gram : mer 
- 7th k-Gram : era 
- 8th k-Gram : rah 
the comparison text used is “bunga biru” (without the quotes) so that the formation of the k-Gram series in the 
comparison text is shown in Figure 3. 


Figure 3. Formation of k-gram on "bunga biru" 


With k = 3 in the text “bunga biru”, seven k-Grams are formed as: 
= [st k-Gram : bun 
- 2nd k-Gram : ung 
+ 3rd k-Gram : nga 
- 4th k-Gram : gab 
- 5th k-Gram : abi 
- 6th k-Gram : bir 
- 7th k-Gram : iru 


3.2. Calculation of hash value 

The hash function is to take string input and convert it to a fixed-length output string [33]. The 
winnowing algorithm uses a rolling hash to calculate the hash value of each gram series. The hash function 
with rolling hash is defined in (1), where H is the hash value, c is the character in grams, b is the base number, 


and k is the number of gram characters: 


Aa = Kb ee Kt eee KO (1) 
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for the second to the last hash value, the calculation does not need to be done using iterations one by one, but 
the hash value calculation can be done using (2). 


Ae2.ck+1 = (Aet.ck Cy x eS) x b+ Cr+1 (2) 


By using the results of the formation of the previous k-Gram series in Text | and Text 2 (comparison 
text), b=2, and k=5, the calculation of the hash value is shown in Table 1. 


Table 1. The result of hash value calculation 
Text Value1 Value2 Value3 Value4 Value5 Value6 Value7 Value8 
Text 1 736 791 895 1115 1547 2424 4161 7642 
Text2 _736 791 895 1104 1529 2388 4109 


3.3. Calculation of hash value 

The winnowing algorithm does not use all the hash values of each gram series that has been formed. 
The hash value formed in the previous step will be divided into a window of size w. The first window contains 
the first hash values up to w. The second window is formed from the second hash value to w+1 and so on until 
a window is formed from all hash values. The window formation from the hash value calculation results in the 
previous stage with the window width (w)=4 is shown in Figure 4 for the Text 1 and Figure 5 for the Text 2. 
The window that is formed in the Text 1 is shown in Table 2. Meanwhile, Table 3 is the result of window 
formation in the Text 2. 


Text1 : 736 791 895 1115. 1547 2424 4161 7642 


Ist Window | 


~ 
a 
Q 


Figure 4. Window formation on text 1 


Text2: 736 791 895 1104 1529 2388 = 4109 


Ist Window | 


a 
4 
= 
B 
a 
2 


Figure 5. Window formation on text 2 


Table 2. The result of window formation on text | 


wil W2 W3 Ww4 
736 791 895 1115 
791 895 1115 1547 
895 1115 1547 2424 
1115 1547 2424 4161 
1547 2424 4161 7642 


Table 3. The result of window formation on text 2 


Wil W2 W3 Ww4 
736 791 895 1104 
791 895 1104 1529 
895 1104 1529 2388 
1104 1529 2388 4109 


3.4. Calculation of fingerprint value 

The fingerprint is a technique that aims to prevent unauthorized copying of digital content, thus 
making digital content less likely to be duplicated [30]. The fingerprint value is determined by selecting the 
smallest hash value from each window. The selection of the fingerprint value from the window formation 
results in the previous stage is shown in Table 4 and Table 5. 
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Table 4. The result of fingerprint value calculation on text 1 


Wil W2 W3 W4 
736 791 895 1115 
791 895 1115 1547 
895 1115 1547 2424 
1115 1547 2424 4161 
1547 2424 4161 7642 


Table 5. The result of fingerprint value calculation on text 2 


Wil W2 W3 W4 
736 791 895 1104 
791 895 1104 1529 
895 1104 1529 2388 
1104 1529 2388 4109 


3.5. Calculation of Jaccard coefficient 

This step is done by calculating the hash value and selecting the smallest fingerprint of the two text 
documents [34]. The fingerprint value formed from the winnowing algorithm is used to measure the percentage 
of similarity of the text using the Jaccard Coefficient, as shown in (3). 


Similarity(d;,d;) = aeraeh x 100% (3) 


In (3), d; and dj are the fingerprint values in the text, w(di) ~ w(dj) is the number of fingerprint values that are 
the same between the i-th text and j-th text, while w(di) U w(dj) is the total fingerprint value of the i-th text and 
the j-th text. The results of the similarity calculation using the Jaccard Coefficient equation for Text 1 and Text 
2 are as: 

Text 1: [736][791][895][1115][1547] 

Text 2: [736][791][895][1104] 


_ 739179111895) 999 
~ [739][791][895][1115][1547][1104] st GO% 


= : xX 100% 
= 50% 


Similarity(d,, d;) 


based on the results of these calculations, it is known that the percentage of text similarity between Text 1 and 
Text 2 is 50%. 


4. RESULTS AND DISCUSSION 

In this study, two testing mechanisms were carried out, which consisted of testing the combination of 
k-gram values, windows, and prime numbers and testing the accuracy of the system assessment results based 
on the teacher's assessment. The first test aims to determine the combination of k-gram values, windows, and 
prime numbers to obtain the right combination to produce good results. The second test aims to measure the 
accuracy of the results of the assessment system with the assessment carried out by the teacher manually. 


4.1. Testing the combination of k-gram values, window, and prime numbers 
The testing mechanism is carried out by comparing the similarity of the two answers. The question 
and answers used in this test are: 

- Question: “Explain the meaning of metamorphosis?” 

- First answer: “Metamorphosis is a biological developmental process in animals that involves changes in 
physical appearance and structure after birth or hatching. These physical changes occur as a result of cell 
growth and cell differentiation which are radically different.” 

- Second answer: “Metamorphosis is the process of a biological development of animals with changes in 
appearance and structure after birth or hatching. These physical changes are caused by cell proliferation 
and differentiation of very different cells.” 

By using ten combinations of k-gram values, windows, and prime numbers, the test results are shown 
in Table 6 and Figure 6. The combination of parameter values in this test was adopted from several previous 
studies [35]-[37], which resulted in a fairly high accuracy for similar cases. These values are used in this study 
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to measure their performance in the case of automatic assessment of student answers, with some adjustment of 
parameter values. 


Table 6. The results of the combination of parameters on one question 


Experiment No. _k-gram | Window _ Prime numbers Similarity percentage 
1 3 4 7 56.54% 
2 3 3 7 57.31% 
3 4 3 7 49.63% 
4 4 7 3 44.57% 
5 6 7 3 35.74% 
6 2 4 7 74.04% 
7 2, 7 is 75.22% 
8 2 7 19 72.96% 
9 4 5 5 47.08% 
10 4 7 19 48.33% 
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Figure 6. Graph of parameter combination test results 


Separately, the test was carried out again on ten essay questions with various combinations of 
parameters shown in Table 7. While the test results are shown in Figure 7. Based on the test results shown in 
Figure 6 and Figure 7 using various combinations of parameters, it is known that the smaller the k-gram value, 
the greater the percentage value, which indicates the high similarity of the text. A small k-gram value indicates 
the more detailed the checking is carried out on each word. The k-gram value functions as a word splitter into 
a number according to the k-gram value. However, too small a k-gram value causes the duration of checking 
each word or sentence to be longer because more and more calculations are carried out so that it can increase 
the computation time. 

The window value also affects the percentage of text similarity. The longer the text and the smaller 
the k-gram, the greater the percentage of similarity of the text. Meanwhile, the larger the prime number, the 
less different the result. 


Table 7. Combination of k-gram values, window, and prime numbers 


Parameter Experiment 1 __ Experiment 2.___ Experiment 3 _ Experiment 4 __ Experiment 5 
k-gram 3 3 4+ 4 3 
window + 4 4+ 3 2 
prime numbers 2 5 5 =) 3 


Indonesian J Elec Eng & Comp Sci, Vol. 29, No. 1, January 2023: 572-582 


Indonesian J Elec Eng & Comp Sci ISSN: 2502-4752 im) 579 


100.00% 


f] lq ig ie 
90.00% | \Egaa : Af] ‘ ‘ 
io oh ; K eho ‘ - 4 
80.00% > ; i Ig aod | ‘ | 
oh : f ; MS B4o He 4 
x 70.00% |; | : Mize ad) ‘= 7 + 
5 Feod | EE ayn EE ER 
2 60.00% -AeAvn | ; /=8 4a oo EB | 
5 Ea4n : a: fof Ee AB ty 
& so00% HEeH }! > a fo ER Ars 
+ 50.00% (BEve d a=: 4g ER Viemme 
2 ee ee ee ne a fa 
& 40.00% EY : " Mi). ag ty ER “gt! 
c= a= ce : EB Ac Ee 4 +] 
S 30.00% -Eayvo -) : Met: “O° ER Ane 
n M/s i 8 ao bi Ee ae 
20.00% ERG + EE rs = ros 
= oe NS cease Comes =F oe 
10.00% —>\Berr! || , ai: ‘ee EB go 
+ + a as: oe + +/8 + 
a= Id » > os Ws 4 BR BY +] 
0.00% a a: = 3 

1 2 3 4 5 6 7 8 9 10 

Question Number 


Experiment 1 OExperiment2 Experiment3 G&Experiment4 OExperiment 5 


Figure 7. The results of testing the combination of k-gram values, window, and prime numbers 


4.2. Accuracy testing based on teacher's assessment 

The evaluation is carried out by comparing student answers with the teacher's answer key. This test 
uses 20 answers, consisting of ten student answers from working on Indonesian language questions and ten 
teacher answer keys. The test results are shown in Table 8. 


Table 8. The result of accuracy test 
No _StudentID _ Assessment by system Assessment by teacher 


1 201#3 63 56 
2 202#2 92 90 
3 202#1 100 90 
4 201#8 68 65 
5 201#4 87 82 
6 202#3 80 78 
7 202#5 67 67 
8 200#3 58 50 
9 200#6 87 719 
10 202#4 87 80 

Total 789 737 


Based on the results of the accuracy-test in Table 8, it is known that the difference between the 
assessment system and the teacher's assessment is 52. Thus, it can be concluded that the level of accuracy 
produced is quite good, with an average difference of 5.2%. Winnowing algorithm is good enough to be 
implemented in short essay questions and still stable for long essay answers. 


4.3. Comparison of winnowing with other algorithms 

Through the test results, using the value of k-gram=3, the winnowing algorithm can provide a 
similarity level of 83.67%. The results showed that the student's answer correction system was running well 
and gave the desired results. When compared with the Robin Karp algorithm implemented in previous studies 
for string matching, the similarity level is 71.13% when using k-gram=3 [38]. The Cosine Similarity algorithm 
has also been used to check document similarity by producing a similarity level of 23% [39]. While the Knuth 
Morris Pratt algorithm provides an accuracy value of 9.20% to detect the similarity of two documents [25]. 
The comparison of the accuracy of the winnowing algorithm and several other algorithms used to check the 
similarity of the two documents can be seen in Table 9. The results of this test prove that the winnowing 
algorithm is able to detect the level of text similarity with a higher level of accuracy than the other three 
algorithms. This is because the winnowing algorithm is quite stable and effective as string matching. 
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Table 9. Comparison of algorithm accuracy 


Algorithm Accuracy 
Winnowing 83.67% 
Rabin Karp 71.13% 
Cosine Similarity 23.00% 
Knuth Morris Pratt 9.20% 


5. CONCLUSION 

The e-learning system was successfully developed using the Laravel framework with several features 
such as logging in, uploading and downloading learning materials, uploading and downloading assignments, 
creating and filling out attendance forms, and creating and working on exam questions. The existence of an 
automatic assessment feature in e-learning is proven to make it easier for teachers to evaluate students’ essay 
questions so that the assessment process can be carried out effectively and efficiently. The winnowing 
algorithm for the assessment process of essay questions can give good performance on short essay answers 
with a hash number not smaller than the window value. 

When used for long essay answers, the winnowing algorithm can still work well with an average 
difference of 5.2% from the actual assessment results. The k-gram value is proven to affect the similarity 
results, where the smaller the k-gram value, the greater the value obtained by students. On the other hand, the 
greater the value of k-gram, the smaller the similarity value, which causes the smaller the value obtained by 
students. In addition, the window value can also affect the same number of fingerprints, where the greater the 
window value, the more hash values are accommodated, so that the more the number of fingerprints are the 
same and can affect the similarity value. The winnowing algorithm is proven to provide the highest accuracy 
value compared to Rabin Karp, Cosine Similarity, and Knuth Morris Pratt when used to detect the level of 
document similarity, with an accuracy value of 83.7%. 
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